SoftTarget Regularization: An Effective Technique to Reduce Over-Fitting in Neural Networks

摘要

Deep neural networks are learning models with a very high capacity andtherefore prone to over-fitting. Many regularization techniques such asDropout, DropConnect, and weight decay all attempt to solve the problem ofover-fitting by reducing the capacity of their respective models (Srivastava etal., 2014), (Wan et al., 2013), (Krogh & Hertz, 1992). In this paper weintroduce a new form of regularization that guides the learning problem in away that reduces over-fitting without sacrificing the capacity of the model.The mistakes that models make in early stages of training carry informationabout the learning problem. By adjusting the labels of the current epoch oftraining through a weighted average of the real labels, and an exponentialaverage of the past soft-targets we achieved a regularization scheme aspowerful as Dropout without necessarily reducing the capacity of the model, andsimplified the complexity of the learning problem. SoftTarget regularizationproved to be an effective tool in various neural network architectures.